Image processing method, image processing apparatus, image pickup apparatus, and non-transitory computer-readable storage medium

ABSTRACT

An image processing method includes the steps of acquiring a first image shot by using a compound eye image pickup apparatus, acquiring image capturing condition information of the first image, acquiring, depending on the image capturing condition information, optical characteristic information of a plurality of optical systems having a plurality of focal lengths different from each other in the compound eye image pickup apparatus, correcting a deterioration of the first image caused by shooting the first image based on the optical characteristic information, and generating a second image based on information of a position and an angle of a ray obtained from the first image whose deterioration is corrected, and the optical characteristic information contains at least one of information related to aberrations and information related to peripheral illumination of the optical systems.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing method of aparallax image.

2. Description of the Related Art

An image pickup apparatus such as a video camera and a digital camerarequires having a thin shape and a high zoom ratio. In a conventionaltypical image pickup system, an optical system is configured bycombining a plurality of optical lenses to suppress generation of anaberration of the optical system and also to satisfy a desired opticalperformance. In order to reduce in size of this optical system,reduction in image size and reduction in diameter of the optical systemare considered. However, it is difficult to reduce the image size whilethe resolution is maintained.

On the other hand, a compound eye image pickup apparatus (compound eyeimage pickup apparatus) which divides an optical system into a pluralityof optical systems to achieve small-size optical systems has beenproposed. The term “compound eye” means a configuration using astructure of an eye of an insect, and for example a configuration inwhich the optical system is constituted by a lens array including aplurality of lens units and each lens unit is reduced in size andreduced in focal length to miniaturize the optical system is known.However, in the conventional compound eye image pickup apparatus, it isdifficult to add an optical zoom function to make a shooting angle ofview variable. This is because the optical zoom function which makes theshooting angle of view variable by using a method of moving a positionof a lens constituting the optical system needs a mechanical movingmechanism and accordingly the image pickup system is increased in size.

Japanese Patent Laid-open No. 2005-303694 discloses a configuration inwhich a lens unit with a short focus and a lens unit with a long focuswhich have angles of view different from each other are disposed tocapture an image to include the same part of an object. In other words,a zoom-up image obtained by an image pickup element corresponding to thelong focus lens is fitted into part of a wide image obtained by an imagepickup element corresponding to the short focus lens. Accordingly, animage which has a high resolution for part of an area of the image andhas a low resolution and a wide angle of view for the other area of theimage can be obtained. This is effective for a surveillance camera, andfor example a suspicious figure at a center area or the like can bezoomed up to be monitored in detail while understanding an entire areaof a monitoring area.

Japanese Patent Laid-open No. 2011-135359 discloses a compound eye imagepickup system which is configured by a plurality of sub-camera modulesin which color filters different from each other are disposed. Inaddition, Japanese Patent Laid-open No. 2011-135359 discloses a methodof performing restoration processing on an image for each colorcomponent to correct an aberration of an optical system which easilyoccurs as the image pickup system is thinned.

The compound eye image pickup apparatus is capable of acquiring the“Light Field” by disposing image pickup systems in an array to obtainparallax images. The “Light Field” means information of a position andan angle of a ray from an object taken into the image pickup apparatus.In detail, the “Light Field” is described in “Ren.Ng, etc., ‘Light FieldPhotography with a Hand-held Plenoptic Camera’, Stanford Tech ReportCTSR 2005-2”. A plurality of configurations of image pickup apparatuswhich are capable of acquiring the “Light Field” are known, and thecompound eye image pickup apparatus is one of them. The “Light Field”obtained as an image on an image pickup element is a parallax image oran image corresponding to a pupil of an optical system according to theconfiguration of the image pickup apparatus, and it is the same in thatthe ray passing through the pupil of the optical system is separatedinto the position and the angle to be obtained. That is, the parallaximage and the image corresponding to the pupil of the optical system canbe treated to be approximately equivalent by rearranging information ofeach pixel. When the “Light Field” is acquired and the image isreconstructed by image processing, focusing and depth adjustment withina predetermined range can be performed after shooting (i.e., capturing)the image. Such a function is beneficial to a thin compound eye imagepickup apparatus to reduce drive units.

However, in the configuration of Japanese Patent Laid-open No.2005-303694, optical axes of the optical systems with focal lengthsdifferent from each other are displaced from each other. Therefore, itis not appropriate for performing continuous zooming in which viewpointsare constant from a wide-angle side to a telephoto side similarly to atypical monocular zoom optical system. Furthermore, in thisconfiguration, “Light Field” cannot be acquired.

In an actual shooting lens, not a little aberration exists. A spread oflight beams at a point which is conjugate to one point in an objectspace is called a point spread function (PSF). A color bleeding, such asan axial chromatic aberration, a spherical aberration of a color, and acoma aberration of the color, in a color image, and a color shift in alateral direction, that is, a chromatic aberration of magnification cancorrespond to a difference in position or shape of the PSF for eachwavelength. When an aberration occurs in each optical system in thecompound eye image pickup apparatus, a reconstructed image isdeteriorated because an image formed via each optical system isdeteriorated. The method disclosed in Japanese Patent Laid-open No.2011-135359 performs the restoration processing on an image formed viaeach of the optical systems in which color filters different from eachother are disposed. However, since a plurality of optical systems withthe same focal length are disposed, zooming cannot be performed.Furthermore, the color filters are different from each other althoughthe plurality of optical systems with the same focal length aredisposed, and accordingly the “Light Field” cannot be acquired.

SUMMARY OF THE INVENTION

The present invention provides an image processing method capable ofgenerating a high-definition image from a shot image obtained via aplurality of optical systems having a plurality of focal lengthsdifferent from each other, an image processing apparatus, an imagepickup apparatus, and a non-transitory computer-readable storage medium.

An image processing method as one aspect of the present inventionincludes the steps of acquiring a first image shot by using a compoundeye image pickup apparatus, acquiring image capturing conditioninformation of the first image, acquiring, depending on the imagecapturing condition information, optical characteristic information of aplurality of optical systems having a plurality of focal lengthsdifferent from each other in the compound eye image pickup apparatus,correcting a deterioration of the first image caused by shooting thefirst image based on the optical characteristic information, andgenerating a second image based on information of a position and anangle of a ray obtained from the first image whose deterioration iscorrected, and the optical characteristic information contains at leastone of information related to aberrations and information related toperipheral illumination of the optical systems.

An image processing apparatus as another aspect of the present inventionincludes an image acquisition unit configured to acquire a first imageshot by using a compound eye image pickup apparatus, an image capturingcondition acquisition unit configured to acquire image capturingcondition information of the first image, an optical characteristicacquisition unit configured to acquire, depending on the image capturingcondition information, optical characteristic information of a pluralityof optical systems having a plurality of focal lengths different fromeach other in the compound eye image pickup apparatus, a correction unitconfigured to correct a deterioration of the first image caused byshooting the first image based on the optical characteristicinformation, and an image processing unit configured to generate asecond image based on information of a position and an angle of a rayobtained from the first image whose deterioration is corrected, and theoptical characteristic information contains at least one of informationrelated to aberrations and information related to peripheralillumination of the optical systems.

An image pickup apparatus as another aspect of the present inventionincludes a plurality of optical systems having a plurality of focallengths different from each other, a plurality of image pickup elementsprovided corresponding to the respective optical systems, an imageacquisition unit configured to acquire a first image shot by using theoptical systems and the image pickup elements, an image capturingcondition acquisition unit configured to acquire image capturingcondition information of the first image, an optical characteristicacquisition unit configured to acquire, depending on the image capturingcondition information, optical characteristic information of the opticalsystems, a correction unit configured to correct a deterioration of thefirst image caused by shooting the first image based on the opticalcharacteristic information, and an image processing unit configured togenerate a second image based on information of a position and an angleof a ray obtained from the first image whose deterioration is corrected,and the optical characteristic information contains at least one ofinformation related to aberrations and information related to peripheralillumination of the optical systems.

A non-transitory computer-readable storage medium as another aspect ofthe present invention stores an image processing program which causes acomputer to execute a process including the steps of acquiring a firstimage shot by using a compound eye image pickup apparatus, acquiringimage capturing condition information of the first image, acquiring,depending on the image capturing condition information, opticalcharacteristic information of a plurality of optical systems having aplurality of focal lengths different from each other in the compound eyeimage pickup apparatus, correcting a deterioration of the first imagecaused by shooting the first image based on the optical characteristicinformation, and generating a second image based on information of aposition and an angle of a ray obtained from the first image whosedeterioration is corrected, and the optical characteristic informationcontains at least one of information related to aberrations andinformation related to peripheral illumination of the optical systems.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an image pickup apparatus in Embodiment1.

FIG. 2 is an explanatory diagram of an image restoration filter in eachembodiment.

FIG. 3 is an explanatory diagram of the image restoration filter in eachembodiment.

FIGS. 4A and 4B are explanatory diagrams of correction state of a pointimage in each embodiment.

FIGS. 5A and 5B are explanatory diagrams of an amplitude component and aphase component in each embodiment.

FIG. 6 is an explanatory diagram of a lens unit (a plurality of opticalsystems) in Embodiment 1.

FIG. 7 is a diagram of an arrangement of the plurality of opticalsystems in Embodiment 1.

FIGS. 8A to 8D are diagrams of the arrangement of the plurality ofoptical systems in Embodiment 1.

FIG. 9 is an explanatory diagram of electronic zooming in Embodiment 1.

FIG. 10 is a block diagram of an image pickup apparatus in Embodiment 1.

FIG. 11 is a block diagram of an image pickup unit in Embodiment 1.

FIG. 12 is a flowchart of a compound eye image processing in Embodiment1.

FIG. 13 is an explanatory diagram of a method of generating areconstructed image in Embodiment 1.

FIG. 14 is an explanatory diagram of the method of generating thereconstructed image in Embodiment 1.

FIG. 15 is an explanatory diagram of the method of generating thereconstructed image in Embodiment 1.

FIG. 16 is an explanatory diagram of an image format in Embodiment 1.

FIG. 17 is an explanatory diagram of electronic zooming in Embodiment 2.

FIG. 18 is an explanatory diagram of the electronic zooming inEmbodiment 2.

FIG. 19 is an explanatory diagram of an image synthesis during theelectronic zooming in Embodiment 2.

FIG. 20 is an explanatory diagram of the image synthesis during theelectronic zooming in Embodiment 2.

FIG. 21 is a block diagram of an image processing system in Embodiment3.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

First of all, the definitions of terms and image restoration processing(image processing method) which are used in this embodiment will bedescribed. The image processing method described in this embodiment isused as appropriate in each embodiment described below.

(Input Image)

An input image is a digital image (a shot image) obtained by receivinglight with an image pickup element via an image pickup optical system,and is deteriorated by an optical transfer function (OTF) depending onthe image pickup optical system and various optical filters. Inaddition, distortion of an image caused by a distortion aberration and acolor shift of an image caused by a chromatic aberration ofmagnification occur. The distortion aberration and the chromaticaberration of magnification can be contained in the optical transferfunction, but in this embodiment they are described separatelyconsidering a case in which correction processing for them is performedindependently. Furthermore, a reduction of a peripheral illumination inwhich a light amount is reduced toward a periphery of an image occurs.The image pickup optical system can use not only a lens, but also amirror (a reflection surface) having a curvature.

The output image has, for example, information on an RGB color componentas a color component. In addition to the RGB color component, a colorspace which is typically used, such as lightness, hue, and chroma whichare expressed by LCH, luminance expressed by YCbCr, and color-differencesignal, can be selected as the color component. As alternative types ofcolor spaces, XYZ, Lab, Yuv, and JCh can also be used. Furthermore, acolor temperature may be used.

The input image or an output image can be accompanied by an imagecapturing condition (image capturing condition information) such as afocal length of a lens, an aperture value, and a shooting distance, andvarious pieces of correction information used to correct the image. Whencorrection processing is performed on an image which is transmitted froman image pickup apparatus to an image processing apparatus, it ispreferred that the image is accompanied by the image capturing conditionor correction information as described above.

(Optical Transfer Function (OTF))

An optical transfer function (OTF) obtained by the Fourier transform ofa point spread function (PSF) is frequency component information on anaberration and is represented by a complex number. An absolute value ofthe OTF, that is, an amplitude component, is referred to as an MTF(Modulation Transfer Function), and a phase component is referred to asa PTF (Phase Transfer Function). Accordingly, the MTF and the PTF arefrequency characteristics of the amplitude component and the phasecomponent of an image deterioration caused by the aberration. The phasecomponent PTF is represented by the following expression (1) as a phaseangle. In expression (1), symbols Re(OTF) and Im(OTF) represent a realpart and an imaginary part of the OTF, respectively.

PTF=tan⁻¹(Im(OTF)/Re(OTF))  (1)

As described above, because the optical transfer function (OTF) of theimage pickup optical system deteriorates the amplitude component and thephase component of an image, with respect to an image at an imagingposition, each point of an object is asymmetrically blurred as seen inthe case of a coma aberration.

(Image Restoration Processing)

A method of correcting a deterioration of the amplitude component (MTF)and a deterioration of the phase component (PTF) of the optical transferfunction (OTF) by using information of the optical transfer function(OTF) of the image pickup optical system is known. This method is calledan image restoration or image recovery, and hereinafter, processing tocorrect the deterioration of the image by using the information of theoptical transfer function (OTF) of the image pickup optical system isreferred to as image restoration processing or restoration processing.

Hereinafter, an outline of the image restoration processing will bedescribed. The following expression (2) is satisfied where g(x,y) is adeteriorated image, f(x,y) is an original image, and h(x,y) is a pointspread function (PSF) which is a Fourier pair of the optical transferfunction (OTF). In expression (2), symbol * denotes a convolution(convolution integration, or convolution sum-product), and symbol (x,y)denotes a coordinate on the image.

g(x,y)=h(x,y)*f(x,y)  (2)

Performing the Fourier transform for expression (2) to convert it into adisplay format on a frequency surface, a product format for eachfrequency is obtained as represented by the following expression (3). Inexpression (3), H is the optical transfer function OTF obtained by theFourier transform of the point spread function PSF (h), G and F arevalues obtained by the Fourier transform of the deteriorated image g andthe original image f, respectively, and (u,v) is a coordinate on atwo-dimensional frequency surface, that is, a frequency.

G(u,v)=H(u,v)·F(u,v)  (3)

In order to obtain the original image f from the deteriorated shotimage, both sides of Expression (3) are divided by the optical transferfunction H, as represented by the following expression (4).

G(u,v)/H(u,v)=F(u,v)  (4)

In expression (4), when the inverse Fourier transform is performed forF(u,v), that is, G(u,v)/H(u,v) to reconvert the frequency surface to areal surface, the original image f(x,y) can be obtained as a restoredimage.

Performing convolution processing on the image on the real surface,similarly, the original image f(x,y) can be obtained as represented bythe following expression (5) where R is a value obtained by the inverseFourier transform of H⁻¹.

g(x,y)*R(x,y)=f(x,y)  (5)

In Expression (5), R (x,y) is called an image restoration filter. Whenthe image is a two-dimensional image, typically, this image restorationfilter is also a two-dimensional filter which has a tap (cell)corresponding to each pixel of the image. The number of taps (number ofcells) of the image restoration filter is set to the number of tapsdepending on required image quality, image processing capability,aberration characteristics, and the like. Since this image restorationfilter needs to reflect at least the aberration characteristics, it isdifferent from a conventional edge-enhanced filter (a high-pass filter)or the like. Since the image restoration filter is determined based onthe optical transfer function (OTF), both of the deteriorations of theamplitude component (MTF) and the phase component (PTF) can be correctedwith high accuracy.

Since an actual image contains a noise component, the use of the imagerestoration filter created by a perfect inverse of the optical transferfunction (OTF) as described above results in a significant amplificationof the noise component along with the restoration of the deterioratedimage. This is because the MTF (amplitude component) of the opticalsystem is boosted such that its value returns to one over allfrequencies from a state in which a noise amplitude is added to theamplitude component of the image. The value of the MTF, which is anamplitude deterioration caused by the optical system, returns to one,but the power spectral of the noise is boosted at the same time, and asa result the noise is amplified depending on the degree of boosting ofthe MTF (i.e., depending on a restoration gain). Thus, if the imagecontains a noise, a satisfactory image as an image to be appreciatedcannot be obtained. This is represented by the following expression(6-1) or (6-2). In expressions (6-1) and (6-2), N is a noise component.

G(u,v)=H(u,v)·F(u,v)+N(u,v)  (6-1)

G(u,v)/H(u,v)=F(u,v)+N(u,v)/H(u,v)  (6-2)

In this respect, a method of controlling the degree of the restorationdepending on the strength ratio SNR (signal-to-noise ratio) of an imagesignal and a noise signal, as in the case of the Wiener filterrepresented by the following expression (7), is known.

$\begin{matrix}{{M\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + {SNR}^{2}}}} & (7)\end{matrix}$

In Expression (7), M(u,v) is frequency characteristics of the Wienerfilter, and |H(u, v)| is an absolute value (MTF) of the optical transferfunction (OTF). This method suppresses the restoration gain (degree ofrestoration) with decrease of the MTF, and it enhances the restorationgain with increase of the MTF for each frequency. Typically, an MTF ofthe image pickup optical system has higher values in low frequencies andlower values in high frequencies, and accordingly the method is a methodof substantially suppressing the restoration gain in the highfrequencies of the image.

Subsequently, referring to FIGS. 2 and 3, the image restoration filterwill be described. FIGS. 2 and 3 are explanatory diagrams of the imagerestoration filter which is used for the image processing method in thisembodiment. The number of taps in the image restoration filter isdetermined depending on aberration characteristics of the image pickupoptical system and on a required restoration accuracy. The imagerestoration filter illustrated in FIG. 2 as an example is atwo-dimensional filter having an 11×11 taps. In FIG. 2, while a value(coefficient) in each tap is omitted, one cross section of the imagerestoration filter is illustrated in FIG. 3. A value (coefficient value)of each tap in the image restoration filter serves as restoring, to anoriginal point, a signal value (PSF) spatially spread due to theaberration. That is, performing convolution processing (convolutionintegration, or convolution sum-product) of the image restoration filteron an image, an image in which the aberration is corrected can beobtained.

Subsequently, referring to FIGS. 4A, 4B, 5A, and 5B, characteristics ofthe image restoration in a real space and a frequency space will bedescribed. FIGS. 4A and 4B are explanatory diagrams of a correctionstate of a point image in this embodiment, and FIGS. 4A and 4Billustrate the PSFs before and after the restoration, respectively.FIGS. 5A and 5B are explanatory diagrams of the amplitude component andthe phase component in this embodiment, respectively. In FIG. 5A, adotted line (1) denotes the MTF before the restoration, and adashed-dotted line (2) denotes the MTF after the restoration. Similarly,in FIG. 5B, a dotted line (1) denotes the PTF before the restoration,and a dashed-dotted line (2) denotes the PTF after the restoration. ThePSF before the restoration is asymmetrically spread, and because of thisasymmetry, the PTF has a value nonlinear with respect to the frequency.The restoration processing corrects the MTF to be amplified and alsocorrects the PTF to be zero, and accordingly the restored PSF issymmetric and sharp-shaped.

With respect to a method of creating the image restoration filter, theimage restoration filter can be obtained by the inverse Fouriertransform of a function designed based on an inverse function of theoptical transfer function (OTF) of the image pickup optical system. Theimage restoration filter used in this embodiment can be changed asappropriate, and the Wiener filter described above, for example, can beused. When the Wiener filter is used, the inverse Fourier transform ofM(u,v) represented by expression (7) allows creation of the imagerestoration filter in a real space which is actually convolved with animage.

The optical transfer function (OTF) varies depending on an image heightof the image pickup optical system (i.e., depending on a position in animage) even in one shooting state. Therefore, it is preferred that theimage restoration filter changes to be used depending on the imageheight. The outline of the image restoration processing is as describedabove.

As image correction processing by using other optical characteristics,distortion correction processing is processing to correct an image basedon distortion information at an imaging position depending on the imageheight. Chromatic aberration of magnification correction processing isprocessing to correct an image based on shift information at the imagingposition for each color component (for example, RGB) depending the imageheight. A peripheral illumination correction processing is processing tocorrect an image based on light amount information depending on theimage height.

Embodiment 1

Next, an image pickup apparatus in Embodiment 1 of the present inventionwill be described. FIG. 1 is a schematic diagram of an image pickupapparatus 100 (camera as a compound eye image pickup apparatus) in thisembodiment. In FIG. 1, reference numeral 101 denotes a camera body(image pickup apparatus body), reference numeral 102 denotes a lens unit(shooting lens unit), reference numeral 103 denotes a lens holder(shooting lens holder), reference numeral 104 denotes a shutter button,reference numeral 105 denotes a zoom lever, and reference numeral 106denotes a grip. The lens unit 102 includes a plurality of opticalsystems (image pickup optical systems). In this embodiment, the lensunit 102 includes 16 optical systems of 4×4 vertically and horizontally,but it is not limited thereto. In this embodiment, the image pickupapparatus 100 includes the lens unit 102 and the camera body 101integrated with each other, but it is not limited thereto. Thisembodiment can also be applied to a configuration in which the lens unit102 (lens apparatus) is removably mounted on the camera body 101.

Subsequently, referring to FIG. 6, a structure of the plurality ofoptical systems in this embodiment will be described. FIG. 6 is anexplanatory diagram of the plurality of optical systems (lens unit 102).The lens unit 102 of this embodiment is constituted by four groups, eachof which includes four optical systems (a) to (d) having four focallengths different from each other, that is, the lens unit 102 includes atotal of 16 optical systems.

The optical system (a) includes, in order from an object side (left sidein FIG. 6), a concave lens 11 a, a convex lens 12 a, a convex lens 13 a,and a concave lens 14 a, and an image pickup element 15 a is disposed onan image plane. Similarly to the optical system (a), the optical systems(b), (c), and (d) include lenses 11 b to 14 b, 11 c to 14 c, and 11 d to14 d, respectively, except for the orders of the concave lenses andconvex lenses. Symbols Oa, Ob, Oc, and Od are optical axes of theoptical systems (a) to (d), respectively. A light amount adjustmentelement 17 is disposed in front of four image pickup elements 15 a to 15d. The light amount adjustment element 17 is an integrally-formedelectroluminescence filter or the like. The plurality of optical systems(a) to (d) are disposed to be close to each other. Accordingly, it ispreferred that an aperture stop or a light shielding wall (notillustrated) which limits an effective range of a ray is providedbetween the lenses and between the lens and the image pickup element soas to prevent unnecessary light from the adjacent optical system fromreaching the corresponding image pickup element 15 a to 15 d. In FIG. 6,symbol F denotes a focus moving unit (focus lens), and it is capable offocusing (i.e., performing focus control) by using a drive mechanism(not illustrated) which integrally drives (moves) the focus moving unitF.

FIGS. 7 and 8A to 8D are diagrams of an arrangement of the plurality ofoptical systems in the lens unit 102. In FIGS. 7 and 8A to 8D, symbols“a” to “d” in respective circles denote, with respect to the 16 opticalsystems constituting the lens unit 102, optical systems each of which isconstituted by a unit (group) having four focal lengths different fromeach other. A point O denotes a centroid (center) position of the 16optical systems. As illustrated in FIG. 8A, the centroid position of theoptical axes Oa of the four optical systems (a) is an optical-axiscentroid Oa2. Similarly, with respect to the optical systems (b), (c),and (d), as illustrated in FIGS. 8B to 8D, the centroids of the opticalaxes Ob, Oc, and Od of the four optical systems (b), (c), and (d) areoptical-axis centroids Ob2, Oc2, and Od2, respectively.

Subsequently, referring to FIG. 9, a method of performing electroniczooming by using the optical systems (a) to (d) having the focal lengthsdifferent from each other will be described. FIG. 9 is an explanatorydiagram of the electronic zooming. As illustrated in FIG. 9, the focallengths of the optical systems (a) to (d) are set to 20 mm, 40 mm, 80mm, and 160 mm, respectively. An image with a focal length 20 mmcorresponding to a viewpoint O is reconstructed by using an imageobtained via the four optical systems (a) illustrated in FIG. 8A. Thus,an image with another viewpoint O which is generated based on imageswith a plurality of viewpoints is called a free viewpoint image. Thatis, the free viewpoint image is an image with a predetermined viewpoint(viewpoint O) which is obtained from a plurality of images with aplurality of viewpoints (i.e., images with an identical angle of viewand also with viewpoints different from each other). A method ofgenerating the free viewpoint image will be described below, along withthe compound eye image processing.

An image with a focal length from 20 mm to 40 mm can be generate bycutting (trimming) the image obtained via the optical system (a). It ispreferred that an image interpolation is used to keep the number ofpixels constant when cutting the image. Similarly, an image with a focallength from 40 mm to 80 mm, an image with a focal length from 80 mm to160 mm, and an image with a focal length from 160 mm to 320 mm can begenerated by cutting (trimming) the images obtained via the opticalsystems (b), (c), and (d), respectively. In this case, aligning theviewpoints (free viewpoints) of the images of respective focal lengthswith the viewpoint O, a virtual optical axis during the electroniczooming is fixed. As described above, the free viewpoint is set to bethe same position with respect to each focal length to fix the opticalaxis during the zooming.

Next, referring to FIG. 10, an internal configuration of the imagepickup apparatus 100 in this embodiment will be described. FIG. 10 is ablock diagram of the image pickup apparatus 100. In FIG. 10, referencenumeral 21 denotes a system control unit which controls a shootingoperation, reference numeral 22 denotes an image pickup unit, referencenumeral 23 denotes a transfer unit, reference numeral 43 denotes animage repairing unit, reference numeral 24 denotes a development unit,and reference numeral 25 denotes an image processing unit. Referencenumeral 26 denotes a compound eye image processing unit, and itcorresponds to an image processing apparatus in this embodiment. Thecompound eye image processing unit 26 includes an image acquisition unit26 a, an image capturing condition acquisition unit 26 b, an opticalcharacteristic acquisition unit 26 c, a correction unit 26 d, and animage processing unit 26 e. Reference numeral 27 denotes a monitor imageprocessing unit, reference numeral 28 denotes a distance calculationunit, and reference 29 denotes an AF/AE evaluation unit. Referencenumeral 30 denotes an operation unit to be operated by a userinstructing a shooting, reference numeral 31 denotes a drive controlunit, reference numeral 32 denotes a driver, reference numeral 33denotes an encoder, reference numeral 34 denotes an exposure controlunit, reference numeral 35 denotes a recording encode unit, referencenumeral 36 denotes a recording unit, reference numeral 37 denotes acommunicating encode unit, reference numeral 38 denotes a communicationunit, and reference numeral 39 denotes an output unit. Reference numeral40 denotes a camera shake detection unit and reference numeral 42denotes a monitor. Hereinafter, functions of respective parts will bedescribed.

(Shooting)

The system control unit 21 includes a CPU and a ROM (not illustrated)which stores a control program to be executed by the CPU, and itcontrols entire processing of the camera (image pickup apparatus 100).The operation unit 30 includes input devices such as a key and a buttonto be used by a user to give a desired instruction to the camera. Inaccordance with the operation of these input devices, the system controlunit 21 supplies, to the monitor 42, data to display a mode transfer ofthe image pickup apparatus 100, a menu screen, or various information.Thus, the monitor 42 displays the data supplied from the system controlunit 21, along with a shot/playback image. The shooting of the image isperformed (i.e., the image is captured) based on a shooting instructionvia the operation unit 30 by the user. When the shooting instruction isgiven, the system control unit 21 sends a shooting instruction signal tothe image pickup unit 22.

Subsequently, referring to FIG. 11, the configuration of the operationof the image pickup unit 22 will be described in detail. FIG. 11 is ablock diagram of the image pickup unit 22. The image pickup unit 22includes the optical systems (a) to (d) illustrated in FIG. 6, the lightamount adjustment element 17, the image pickup element 15 (image pickupelements 15 a to 15 d) provided corresponding to the respective opticalsystems, and an A/D converter 41. The shooting instruction signal fromthe system control unit 21 is sent to the image pickup element 15 (eachof the image pickup elements 15 a to 15 d). The image pickup element 15performs exposure in a predetermined period of time to generate chargesby photoelectric conversion. The image pickup element 15 is for examplea CMOS image sensor, which reads out an image pickup signal (analogsignal) at a predetermined readout timing by a rolling shutter method.The A/D converter 41 converts the analog signal read from the imagepickup element 15 to a digital data row (digital signal), and thensupplies the digital signal to the transfer unit 23.

With respect to the image pickup signal (digital signal) from theplurality of optical systems (a) to (d), the transfer unit 23appropriately adjusts the order and the timing of the output of data ofthe image pickup signal depending on the configuration of a processorsuch as the development unit 24 and the image processing unit 25 toperform processing at the latter stages. The image repairing unit 43performs image processing to repair (or reduce) a noise or defect foreach pixel, and then it supplies data as so-called RAW data to thedevelopment unit 24. The development unit 24 performs colorinterpolation processing for each pixel to generate an image signal, andit supplies a digital image pickup signal within a predetermined dynamicrange to the image processing unit 25. In the color interpolationprocessing, the development unit 24 performs color decode processing sothat RGB information is allocated to all pixels corresponding torespective color filter structures of the image pickup elements 15 a to15 d (image sensor).

When the shutter of the operation unit 30 is pressed, the system controlunit 21 outputs an instruction of the image processing to the imageprocessing unit 25. Then, the image processing unit 25 performs theimage processing such as white balance correction, gamma correction,sharpness correction, and chroma correction.

(Compound Eye Image Processing)

The compound eye image processing unit 26 (image processing apparatus)generates an RGB image (reconstructed image signal) shot with a virtualoptical axis of a shot angle of view instructed by a user via theoperation unit 30 (zoom lever 105 in FIG. 1). The virtual optical axismeans an optical axis which is determined when a virtual lens system isset at the centroid O of the viewpoint illustrated in FIG. 7. In thisembodiment, the compound eye image processing unit 26 generates theimage signal (reconstructed image signal) in which the viewpoint ismoved (corrected) to the virtual optical axis O based on images withfour viewpoints. When the shot angle of view designated by the user isdifferent from the focal lengths of the optical systems (a) to (d), thecompound eye image processing unit 26 performs electronic zooming to cutout an area corresponding to the designated angle of view from thereconstructed image signal shot via the optical system with the focallength one step wider (with one step wider angle) than the designatedangle of view.

Next, referring to FIG. 12, an image processing method (compound eyeimage processing) performed by the compound eye image processing unit 26will be described. FIG. 12 is a flowchart of the compound eye imageprocessing. Each step in FIG. 12 is performed mainly by the compound eyeimage processing unit 26 based on an instruction of the system controlunit 21.

First, at step S11, the compound eye image processing unit 26 (imageacquisition unit 26 a) acquires a multi-viewpoint image (first image).Subsequently, at step S12, the compound eye image processing unit 26(image capturing condition acquisition unit 26 b) acquires an imagecapturing condition (image capturing condition information) of the imagepickup apparatus 100. The image capturing condition is for example azoom position, an aperture value (F number), and a shooting distance.When the image pickup apparatus 100 does not include for example anaperture stop which is not provided with a variable opening, theaperture value may be omitted from the image capturing condition. Thecompound eye image processing unit 26 can acquire the image capturingcondition from information added to the multi-viewpoint image acquiredat step S11. Alternatively, the compound eye image processing unit 26may acquire the image capturing condition directly from the image pickupapparatus 100 (system control unit 21). In this case, the compound eyeimage processing unit 26 only has to perform the processing at step S11before starting step S14.

Subsequently, at step S13, the compound eye image processing unit 26(optical characteristic acquisition unit 26 c) acquires apreviously-stored optical characteristic (optical characteristicinformation) for each of the plurality of optical systems (a) to (d).The optical characteristic means a characteristic related to anaberration of the optical system, and it includes at least one of achromatic aberration of magnification (chromatic aberration ofmagnification correction information), a distortion (distortioncorrection information), an optical transfer function, and an imagerestoration filter generated based on the optical transfer function. Thecompound eye image processing unit 26 is capable of acquiring a motionblur locus (motion blur information such as a motion blur function) ofthe image pickup apparatus 100 during the exposure by using the camerashake detection unit 40 including a gyro sensor or the like. The opticalcharacteristic can includes the motion blur function or a motion blurcorrection filter generated based on the motion blur function to correcta motion blur. The compound eye image processing unit 26 treats, as aPSF, the motion blur locus which occurs according to the shooting of theimage pickup apparatus 100 held by hand or the like, and thus the imagerestoration processing can be applied. The optical characteristic mayfurther include characteristics of a peripheral illumination (peripheralillumination correction information). Data related to the opticalcharacteristic (optical characteristic information) can be selectedamong from data previously stored in the image pickup apparatus 100, andthe previously-stored data may be corrected depending on the imagecapturing condition. The optical characteristic such as a motion blurcan be acquired when shooting an image.

Subsequently, at step S14, the compound eye image processing unit 26(correction unit 26 d) performs correction processing on the image(first image) shot via each optical system by using the opticalcharacteristic acquired at step S13. The correction processing includeschromatic aberration of magnification correction processing whichcorrects a difference of the shot magnification between RGB, distortioncorrection processing which corrects distortion of an image, imagerestoration processing which corrects a deterioration by the opticaltransfer function, and the like. The correction processing may includeperipheral illumination correction processing which corrects a shortageof a light amount at the periphery of an image. Subsequently, at stepS15, the compound eye image processing unit 26 (image processing unit 26e) performs image reconstruction processing on the multi-viewpoint imagein which the decrease of an optical image quality is corrected (i.e.,corrected first image). Then, at step S16, the compound eye imageprocessing unit 26 outputs a reconstructed image (second image).

In order to reduce the load of the processing, it is preferred that theacquisition of the optical characteristic at step S13 and the correctionprocessing at step S14 are performed only on an image necessary for thereconstruction processing of the output image. For example, in FIG. 9,since an image obtained via the optical system (c) may be used in orderto output an image with a focal length of 100 mm, the correctionprocessing only has to be performed on the image obtained via the fouroptical systems (c) in FIG. 7. For images to be used for thereconstruction processing, a high-definition image which does notrequire the correction may be excluded from images to be corrected.

Next, referring to FIGS. 13 to 15, a method of generating thereconstructed image (second image) by the compound eye image processingapparatus 26 will be described. FIGS. 13 to 15 are explanatory diagramsof the method of generating the reconstructed image.

In FIG. 13, as described above, an object p is imaged on image pickupelements S1 and S2 (corresponding points p1′ and p2′) via two lenssystems A1 and A2 (optical systems), respectively, and it is read out asan image pickup signal. First, the system control unit 21 (compound eyeimage processing unit 26) extracts the corresponding points p1′ and p2′from images generated via the lens systems A1 and A2, based on acorrelation of a luminance signal distribution or a color signaldistribution of the images.

Subsequently, the system control unit 21 (compound eye image processingunit 26) obtains a distance (object distance L) from a position of aprincipal point of each of the lens systems A1 and A2 to the object p byusing the following expression (8). As illustrated in FIG. 14,p1′(x1′,y1′) and p2′(x2′,y2′) are coordinates with reference to originsof the images via the lens systems A1 and A2, respectively, L is theobject distance, D is a gap between two optical axes o1 and o2, fa′ is adistance from each of the principal points of the lens systems A1 and A2to the respective image pickup element. Furthermore, d1 is a gap betweenthe corresponding point p1′ and an intersection O1 of the optical axiso1 and the image pickup element S1, and d2 is a gap between thecorresponding point p2′ and an intersection O2 of the optical axis o2and the image pickup element S2. The system control unit 21 (compoundeye image processing unit 26) uses these values to obtain a coordinate(x″,y″,L) of an object p″ on computer with reference to an image pickupsystem origin O′ (midpoint of the two lens systems A1 and A2 in FIG.14).

$\begin{matrix}{L = \frac{D \cdot {fa}^{\prime}}{{d\; 1} + {d\; 2}}} & (8)\end{matrix}$

In this embodiment, due to an error of the focal lengths of the lenssystems A1 and A2 or a position shift between the lens systems A1 and A2and the image pickup elements S1 and S2, reverse trace lines (straightlines R1′ and R2′) in FIG. 14 may not intersect with each other on athree-dimensional space on computer in some cases. In this case, avirtual flat plane is set at a position where the two straight lines R1′and R2′ are closest, and a midpoint of this flat plane and theintersection of the straight lines may be obtained as a coordinate. Whenthere are three or more lens systems, similar calculations can beperformed by using straight lines obtained from respective lens systems.

Subsequently, an object p″ on computer, obtained as illustrated in FIG.15, is projected on a virtual image pickup element by using a lenssystem A0 with the focal length fa′ virtually disposed at the imagepickup optical system origin O′, and a coordinate p0′(x0′,y0′) of avirtual image is obtained. The compound eye image processing unit 26performs the image processing for an entire image to generate areconstructed image with the virtual optical axis by using the virtuallens system A0.

The generation processing of the reconstructed image described abovecorresponds to generation of the reconstructed image (second image)based on information of positions and angles of rays obtained from theplurality of parallax images (first images).

(Monitor Display in Shooting)

Next, processing of displaying an image on the monitor 42 constituted bya thin unit such as a liquid crystal panel and an organic EL when thepower of the image pickup apparatus 100 is turned on will be described.The monitor image processing unit 27 receives an output of the compoundeye image processing unit 26, and it displays an image. Similarly to thecompound eye image processing unit 26, the monitor image processing unit27 simply performs correction processing of a viewpoint, and itgenerates a reconstructed image signal for display and sends it to themonitor 42. The monitor 42 displays the reconstructed image signal fordisplay in real time. According to the simple display processing, themonitor image processing unit 27 can display a shot image on the monitor42 while omitting image processing requiring much processing time suchas image cutout processing, interpolation processing, and optical axisalignment processing. Therefore, a screen with a small amount of delayfor display can be provided to a user. Accordingly, a convenient imagepickup apparatus which does not interfere in the framing operation foran object or the like can be achieved. The correction processing byusing the optical characteristic can be performed while making a choiceappropriately in the simple display processing.

(Image Recording)

The recording encode unit 35 encodes the reconstructed image signalgenerated by the compound eye image processing unit 26 into apredetermined format as an example illustrated in FIG. 16. FIG. 16 is anexplanatory diagram of an image format in this embodiment. The imageformat includes a header section of files to store shooting information,an image data section to store image data (image signals a1 to d4), anda distance map data section to store distance map data.

The header section includes “lens designation”, “lens information”,“lens position relation”, “shooting information”, “pixel structure”, and“image format”. The section of the “lens designation” stores informationrelated to a type or structure of the lens system (optical system)provided in the image pickup apparatus 100. The section of the “lensinformation” stores angle of view information and optical characteristicinformation for each lens, and the like. The section of the “lensposition relation” stores information of the position relation of theoptical axis for each lens. Each of the “lens designation”, the “lensinformation”, and the “lens position relation” corresponds to acalculation coefficient. The section of the “shooting information”stores information of the angle of view when a user instructs shooting,information of latitude and longitude of a shooting location,information of time at the shooting location, and information of adirection of the image pickup apparatus 100 in the shooting. The sectionof the “pixel structure” stores information of the number of pixels invertical and horizontal directions for recorded image data. The sectionof the “image format” stores information of a compression of an imageand a type of the compression.

As image data, the reconstructed image signal generated with an angle ofview instructed by the user is recorded, and subsequently, image dataa1, a2, a3, and a4 shot via the optical systems (a) are recorded. Then,the image data with respect to the optical systems (b), (c), and (d) arerecorded similarly. As distance map data, distance maps based on imagesignals (image data) shot via the optical systems (a) to (d) aresequentially recorded. The recording unit 36 records the reconstructedimage signal encoded by the recording encode unit 35 in a recordingmedium such as a memory card (not illustrated).

Embodiment 2

Next, referring to FIGS. 17 to 20, compound eye image processing alongwith electronic zooming in Embodiment 2 of the present invention will bedescribed. FIGS. 17 and 18 are explanatory diagrams of the electroniczooming in this embodiment. FIGS. 19 and 20 are explanatory diagrams ofimage synthesis processing during the electronic zooming in thisembodiment. The basic configuration of an image pickup apparatus in thisembodiment is the same as that of the image pickup apparatus 100 inEmbodiment 1, and accordingly common descriptions are omitted.

In the compound eye image processing in Embodiment 1 described referringto FIG. 9, when an image pickup angle of view designated by a user isdifferent from the focal lengths of the optical systems (a) to (d), theelectronic zooming is performed for a reconstructed image shot by usingan optical system having a focal length at a wide angle side at whichthe angle of view is wider by one step than the designated angle ofview. On the other hand, in this embodiment, as illustrated in FIG. 17,when the designated image pickup angle of view is different from thefocal lengths of the optical systems (a) to (d), the electronic zoomingis performed for reconstructed images shot by using optical systemshaving focal lengths at a wide angle side and a telephoto side at whichthe angles of view are wider and narrower by one step respectively thanthe designated angle of view.

Symbols “a” and “b” in FIG. 19 denote images obtained via the opticalsystems (a) and (b), respectively. When the focal length designated bythe user corresponds to an image X, the images “a” and “b” can besynthesized with respect to an area where the image “b” overlaps in theimage X. On the other hand, with respect to an area where the image “b”does not overlap in the image X, the images “a” and “b” cannot besynthesized, and accordingly, only the image “a” is used. The same istrue for a focal length between the optical systems (b) and (c) orbetween the optical systems (c) and (d).

As illustrated in FIG. 18, all or part of the images “a”, “b”, “c”, and“d” can also be selectively used when an image of for example a focallength between the optical systems (a) and (b) is generated.Accordingly, as illustrated in FIG. 20, high-resolution images (images“c” and “d”) at the telephoto side compared to the image X can besynthesized. The correction processing of an image using the opticalcharacteristics can be performed only for an image used to generate theimage X having a focal length designated by the user to improve thespeed of the processing.

Embodiment 3

Next, referring to FIG. 21, Embodiment 3 of the present invention willbe described. This embodiment describes an image processing system whichperforms the image processing method described above. FIG. 21 is a blockdiagram of the image processing system in this embodiment. Asillustrated in FIG. 21, the image processing system includes an imagepickup apparatus 301. The image pickup apparatus 301 is, for example, acompound eye image pickup apparatus such as a camera, a microscope, anendoscope and a scanner, and it has a basic configuration similar to theimage pickup apparatus 100 of Embodiment 1 described referring to FIG.10. An image processing apparatus 302 is a computer device (informationprocessing apparatus) which is capable of performing the imageprocessing method of each embodiment described above. The imageprocessing apparatus 302 has the same function as that of the compoundeye image processing unit 26 (image acquisition unit 26 a, imagecapturing condition acquisition unit 26 b, optical characteristicacquisition unit 26 c, correction unit 26 d, and image processing unit26 e).

A recording medium 303 is for example a semiconductor memory, a harddisk, or a server on a network, and it stores a shot image. The imageprocessing apparatus 302 acquires shot image data from the image pickupapparatus 301 or the recording medium 303, and then outputs the imagedata for which predetermined processing is performed to at least one ofan output device 305, the image pickup apparatus 301, and the recordingmedium 303. Alternatively, the image data can be output to and stored ina storage unit included in the image processing apparatus 302. Theoutput device 305 is for example a printer. A display device 304(monitor 42) is connected to the image processing apparatus 302 to inputa corrected image to the display device 304. The user can work whileconfirming the corrected image through the display device 304. Imageprocessing software 306 (image processing program) executes the imageprocessing method of each embodiment described above, along withdevelopment processing or other image processing as needed. The displaydevice 304 is for example a liquid crystal display or a projector.

In this embodiment, it is preferred that data such as correctioninformation to perform the image processing method and informationrelated to the transmission and reception of the data between devicesare accompanied by individual image data. Since necessary correctioninformation is accompanied by the image data, the correction processingcan be performed by any device which includes the image processingapparatus of this embodiment.

Other Embodiments

Embodiment (s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD™),a flash memory device, a memory card, and the like.

The image processing method in each embodiment includes a step (stepS11) of acquiring a first image shot by using a compound eye imagepickup apparatus, a step (step S12) of acquiring image capturingcondition information of the first image, and a step (step S13) ofacquiring, depending on the image capturing condition information,optical characteristic information of a plurality of optical systemshaving a plurality of focal lengths different from each other in thecompound eye image pickup apparatus. The image processing method furtherincludes a step (step S14) of correcting the first image (i.e.correcting a deterioration of the first image caused by shooting thefirst image) based on the optical characteristic information, and a step(step S15) of generating a second image based on information of aposition and an angle of a ray obtained from the corrected first image(i.e. the first image whose deterioration is corrected). The opticalcharacteristic information contains at least one of information relatedto aberrations and information related to peripheral illumination of theoptical systems.

Preferably, the focal lengths are fixed. The optical systems include aplurality of first optical systems (one of optical systems (a) to (d))having a first focal length, and a plurality of second optical systems(another of the optical systems (a) to (d)) having a second focal lengthdifferent from the first focal length. More preferably, the opticalcharacteristic information contains at least one of chromatic aberrationof magnification correction information, distortion correctioninformation, and peripheral illumination correction information.Preferably, the optical characteristic information contains at least oneof an optical transfer function and an image restoration filtergenerated based on the optical transfer function. Preferably, theoptical characteristic information contains at least one of a motionblur function and a motion blur correction filter generated based on themotion blur function. Preferably, the optical characteristic informationis different depending on the optical systems.

Preferably, the second image is generated by reconstruction processingon the first image whose deterioration is corrected (i.e. the correctedfirst image). More preferably, the second image is generated byelectronic zooming after the reconstruction processing on the firstimage whose deterioration is corrected. Preferably, the focal lengthsinclude a first focal length at a wide-angle side and a second focallength at a telephoto side. The second image is an image having an angleof view which corresponds to a third focal length between the first andsecond focal lengths. Preferably, the second image is generated by theelectronic zooming after the reconstruction processing on the firstimage having an angle of view which corresponds to the first focallength. Preferably, the second image is generated by the electroniczooming and image synthesis processing after the reconstructionprocessing on the first image having an angle of view which correspondsto each of the first and second focal lengths.

Preferably, in the reconstruction processing on the first image whosedeterioration is corrected, free viewpoint images related to therespective focal lengths are generated by using the first image whosedeterioration is corrected. More preferably, the free viewpoint imagesrelated to the respective focal lengths have viewpoints which coincidewith each other. In other words, the free viewpoints are set to anidentical position with respect to each focal length to fix an opticalaxis during zooming. Preferably, the step of correcting thedeterioration of the first image includes correcting a deterioration ofa specific first image of the first images based on the opticalcharacteristic information of a specific optical system of the opticalsystems.

The present invention can provide an image processing method capable ofgenerating a high-definition image from a shot image obtained via aplurality of optical systems having a plurality of focal lengthsdifferent from each other, an image processing apparatus, an imagepickup apparatus, and a non-transitory computer-readable storage medium.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-157545, filed on Aug. 1, 2014, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. An image processing method comprising the stepsof: acquiring a first image shot by using a compound eye image pickupapparatus; acquiring image capturing condition information of the firstimage; acquiring, depending on the image capturing conditioninformation, optical characteristic information of a plurality ofoptical systems having a plurality of focal lengths different from eachother in the compound eye image pickup apparatus; correcting adeterioration of the first image caused by shooting the first imagebased on the optical characteristic information; and generating a secondimage based on information of a position and an angle of a ray obtainedfrom the first image whose deterioration is corrected, wherein theoptical characteristic information contains at least one of informationrelated to aberrations and information related to peripheralillumination of the optical systems.
 2. The image processing methodaccording to claim 1, wherein the focal lengths are fixed, and whereinthe optical systems include: a plurality of first optical systems havinga first focal length, and a plurality of second optical systems having asecond focal length different from the first focal length.
 3. The imageprocessing method according to claim 1, wherein the opticalcharacteristic information contains at least one of chromatic aberrationof magnification correction information, distortion correctioninformation, and peripheral illumination correction information.
 4. Theimage processing method according to claim 1, wherein the opticalcharacteristic information contains at least one of an optical transferfunction and an image restoration filter generated based on the opticaltransfer function.
 5. The image processing method according to claim 1,wherein the optical characteristic information is different depending onthe optical systems.
 6. The image processing method according to claim1, wherein the second image is generated by reconstruction processing onthe first image whose deterioration is corrected.
 7. The imageprocessing method according to claim 6, wherein the second image isgenerated by electronic zooming after the reconstruction processing onthe first image whose deterioration is corrected.
 8. The imageprocessing method according to claim 7, wherein the focal lengthsinclude a first focal length at a wide-angle side and a second focallength at a telephoto side, and wherein the second image is an imagehaving an angle of view which corresponds to a third focal lengthbetween the first and second focal lengths.
 9. The image processingmethod according to claim 8, wherein the second image is generated bythe electronic zooming after the reconstruction processing on the firstimage having an angle of view which corresponds to the first focallength.
 10. The image processing method according to claim 8, whereinthe second image is generated by the electronic zooming and imagesynthesis processing after the reconstruction processing on the firstimage having an angle of view which corresponds to each of the first andsecond focal lengths.
 11. The image processing method according to claim6, wherein in the reconstruction processing on the first image whosedeterioration is corrected, free viewpoint images related to therespective focal lengths are generated by using the first image whosedeterioration is corrected.
 12. The image processing method according toclaim 11, wherein the free viewpoint images related to the respectivefocal lengths have viewpoints which coincide with each other.
 13. Theimage processing method according to claim 1, wherein the step ofcorrecting the deterioration of the first image includes correcting adeterioration of a specific first image of the first images based on theoptical characteristic information of a specific optical system of theoptical systems.
 14. An image processing apparatus comprising: an imageacquisition unit configured to acquire a first image shot by using acompound eye image pickup apparatus; an image capturing conditionacquisition unit configured to acquire image capturing conditioninformation of the first image; an optical characteristic acquisitionunit configured to acquire, depending on the image capturing conditioninformation, optical characteristic information of a plurality ofoptical systems having a plurality of focal lengths different from eachother in the compound eye image pickup apparatus; a correction unitconfigured to correct a deterioration of the first image caused byshooting the first image based on the optical characteristicinformation; and an image processing unit configured to generate asecond image based on information of a position and an angle of a rayobtained from the first image whose deterioration is corrected, whereinthe optical characteristic information contains at least one ofinformation related to aberrations and information related to peripheralillumination of the optical systems.
 15. An image pickup apparatuscomprising: a plurality of optical systems having a plurality of focallengths different from each other; a plurality of image pickup elementsprovided corresponding to the respective optical systems; an imageacquisition unit configured to acquire a first image shot by using theoptical systems and the image pickup elements; an image capturingcondition acquisition unit configured to acquire image capturingcondition information of the first image; an optical characteristicacquisition unit configured to acquire, depending on the image capturingcondition information, optical characteristic information of the opticalsystems; a correction unit configured to correct a deterioration of thefirst image caused by shooting the first image based on the opticalcharacteristic information; and an image processing unit configured togenerate a second image based on information of a position and an angleof a ray obtained from the first image whose deterioration is corrected,wherein the optical characteristic information contains at least one ofinformation related to aberrations and information related to peripheralillumination of the optical systems.
 16. A non-transitorycomputer-readable storage medium storing an image processing programwhich causes a computer to execute a process comprising the steps of:acquiring a first image shot by using a compound eye image pickupapparatus; acquiring image capturing condition information of the firstimage; acquiring, depending on the image capturing conditioninformation, optical characteristic information of a plurality ofoptical systems having a plurality of focal lengths different from eachother in the compound eye image pickup apparatus; correcting adeterioration of the first image caused by shooting the first imagebased on the optical characteristic information; and generating a secondimage based on information of a position and an angle of a ray obtainedfrom the first image whose deterioration is corrected, wherein theoptical characteristic information contains at least one of informationrelated to aberrations and information related to peripheralillumination of the optical systems.